Behavior of Accelerated Gradient Methods near Critical Points of Nonconvex Problems∗

We examine the behavior of accelerated gradient methods in smooth nonconvex unconstrained optimization. Each of these methods is typically a linear combination of a gradient direction and the previous step. We show by means of the stable manifold theorem that the heavyball method method does not converge to critical points that do not satisfy second-order necessary conditions. We then examine the behavior of two accelerated gradient methods — the heavy-ball method and Nesterov’s method [6] — in the vicinity of the saddle point of a nonconvex quadratic function, showing in both cases that the accelerated gradient method can diverge from this point more rapidly than steepest descent.